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-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 



- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication, 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)M This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) [x] Claim(s) 1-45 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) [x] Ciaim(s) 1-45 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) [x] The drawing(s) filed on 10 December 2001 is/are: a)IEI accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) 13 The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1 ) [El Notice of References Cited (PTO-892) 4) Q Interview Summary (PTO-413) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Pa P er No(s)/Mail Date. . 

3) M Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-1 52) 

Paper No(s)/Mail Date 12/10/01 . 6) □ Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 11012004 
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DETAILED ACTION 



Oath/Declaration 



1 . The oath or declaration is defective. A new oath or declaration in compliance 
with 37 CFR 1 .67(a) identifying this application by application number and filing date is 
required. See MPEP §§ 602.01 and 602.02. 

The oath or declaration is defective because: 

It does not identify the city and either state or foreign country of residence of 
each inventor. The residence information may be provided on either on an 
application data sheet or supplemental oath or declaration. (Note: Only P.O. 
Box information is provided.) 



Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-5, 13-14, 24-29, 37-38 and 45 are rejected under 35 U.S.C. 102(b) as 
being unpatentable over Grimson et al. ("Using Adaptive Tracking to Classify and 



Monitor Activities in a Site," Proceedings of IEEE Conference on Computer Vision and 
Pattern Recognition, 23-25 June 1998, pp. 22-29), in view of Bhat et al. ("Motion 
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Detection and Segmentation Using Image Mosaics," IEEE International Conference on 
Multimedia and Expo, Vol. 3, 30 July -2 Aug. 2000, pp. 1577-1580). 



4. Regarding claim 1 , and similarly claims 25 and 45, Grimson discloses 

• maintaining one or more pixel-level historical models of spatially local 
pixel observations 

[P. 23, left column. Note that the model is based on each pixel's 
observed intensity value.] 

• segmenting pixels into two or more labeled groups based at least in part 
upon comparison of pixel -level video input with the one or more pixel - 
level historical models 

[P. 23: left column, 3 rd line from bottom through line 34 of the right 
column. Note that depending on the matching result, pixels are 
segmented into a foreground group and a background group. In addition, 
P. 26, right column, lines 1-5 disclose further classifying foreground 
(i.e., tracked objects) into labeled groups such as cars, trucks and 
people using spatially non-local characteristics such as aspect ratio 
and size . ] 

• updating the one or more pixel-level historical models t>ased at least in 
part upon one or more feedback maps identifying pixels irespect ively 
segmented into the one or more labeled groups 

[P. 23: left column, 3 rd line from bottom through line 3 0 of the right 
column. Note that the matching operation segments the image into the 
foreground part and the background part (each part obviously can be 
represented by a respective binary map) and updates are carried out 
accordingly. ] 



Grimson does not expressly disclose that the feedback maps for updates identifies 
pixels respectively segmented into labeled groups in conformity with a spatially non- 
local segmentation model. 



However, Bhat teaches/suggests using labeled foreground groups resulted from 
spatially non-local segmentation to update a decision process based on a background 
model. [See P. 1578: Sect. 3, lines 14-17; Sect. 3.1, last paragraph; Sect. 3.2, 1 st 
paragraph. Note that lines 2-4 of Sect. 3.2 indicate that pixels are classified into groups 
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such as "people," which implies the employment of non-local spatial information, as is 
obvious to one of ordinary skill in the art. (See for example, the disclosure by Grimson 
discussed above in which aspect ratio and size are used.)] 

Grimson and Bhat are combinable because they have aspects that are from the same 
field of endeavor of segmentation/classification. 

At the time of the invention, it would have been obvious to one of ordinary skill in the art 
to modify Grimson with the teachings of Bhat by using feedbacks identifying spatially 
non-locally segmented pixel groups to update the pixel-level historical models . The 
motivation would have been to because it is well know in the art that local features such 
as a pixel's intensity is insufficient in most of the region-labeling applications; high-level 
knowledge such as shape and size are needed to achieve more accurate results. 
When improved labeling (e.g., as foreground or background) is done, unnecessary 
adjustment to the models can be avoided. 

Therefore, it would have been obvious to combine Bhat with Crimson to obtain the 
invention of claim 1. 

5. Regarding claim 2, and similarly claim 26, Grimson discloses modeling by a 
mixture of Gaussian distributions. [See P. 23, left column, lines 3-8. Note that this fact 
has also been admitted in P. 1, lines 11-12 of the application.] 
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6. Regarding claim 3, and similarly claim 27, Grimson discloses segmenting the 
pixels into a foreground group and a background group. [See P. 23: left column, 3 rd line 
from bottom through line 34 of the right column of Grimson. In addition, Bhat also has 
the same disclosure. See P. 1578, Sect. 3.1, lines 8-10 of Bhat.] 



7. Regarding claim 4, and similarly claims 7, 28 and 31 , Bhat further discloses 

• wherein a feedback map identifies pixels segmented correctly according 
to a spatially non-local segmentation model 

[Sect. 3.2, 1 st paragraph, especially lines 11-15. Note that the 
correctly and incorrectly classified pixels determine "positive" 
feedback map(s) and "negative" feedback map(s) (in the sense as 
described in the application), respectively.] 



8. Regarding claim 5, and similarly claim 29, they are rejected as per the analysis of 
claim 1, specifically 

• wherein the spatially non-local segmentation model defines spatially 
non-local observation characteristics of pixels belonging to one of the 
labeled groups 

[Per the analysis of claim 1, non-local features are used to classify 
pixels into labeled groups such as people, cars and trucks, etc.] 



9. Regarding claims 13-14, and similarly claims 37-38, Bhat discloses 

• (claim 13) generating the feedback maps based at least in part upon 
analysis of spatially non-local video frame features 
and 

(claim 14) wherein the feedback maps are generated based at least in 
part upon one or more of an image region analysis, a frame-wide image 
statistics analysis, or an analysis of the object or event content of 
the video frames 

[See P . 1578: Sect. 3, lines 14-17; Sect. 3.1, last paragraph; Sect. 
3.2, 1 st paragraph. Note that lines 2-4 of Sect. 3.2 indicate that 
pixels are classified into group such as "people," which implies the 
employment of non-local spatial information, as is obvious to one of 
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ordinary skill in the art. (See for example, the disclosure by Grimson 
discussed above.)] 



10. Regarding claim 24, Bhat discloses 

• wherein one or more feedback maps are generated by one or more of a 

person detector and tracker module, a rapid illumination change detector 
module, a camera gain change detector module, or a sudden camera motion 
detector module 

[P. 1578, Sect. 3.2, lines 2-7 where people or person are determined 
and labeled] 



*** 

1 1 . Claims 6-9 and 30-33 are rejected under 35 U.S.C. 102(b) as being unpatentable 
over Grimson et al. (Using Adaptive Tracking to Classify and Monitor Activities in a 
Site," Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 
23-25 June 1998, pp. 22-29]) and Bhat et al. ("Motion Detection and Segmentation 
Using Image Mosaics," IEEE International Conference on Multimedia and Expo, Vol. 3, 
30 July -2 Aug. 2000, pp. 1577-1580) as applied to claims1-5, 13-14, 24-30, 37-38 and 
45, and further in view of Javidi (US 5,699,449). 

12. Regarding claim 6, and similarly claim 30, the combined invention of Grimson 
and Bhat discloses all limitations of its parent, claim 1 . 

The combined invention of Grimson and Bhat does not expressly disclose 

• wherein a pixel -level historical model is not updated at pixels 
identified as being correctly segmented 
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However, Javidi teaches/suggests that updates are not performed at pixels correctly 
classified. [See Col. 6, lines 15-55, especially lines 45-47.] 

The combined invention of Grimson and Bhat is combinable with Javidi because they 
have aspects that are from the same field of endeavor of segmentation/classification. 

At the time of the invention, it would have been obvious to one of ordinary skill in the art 
to modify the combined invention of Grimson and Bhat with the teachings of Javidi by 
not performing model update if the pixels in question have been correctly segmented . 
The motivation would have been that successful segmentation indicates the "fitness" of 
the model, at least for those pixels; therefore no update is warranted. 

Therefore, it would have been obvious to combine Javidi with Grimson and Bhat to 
obtain the invention of claim 6. 

1 3. Regarding claim 7, and similarly claim 31 , Bhat further discloses 

• wherein a feedback map identifies pixels segmented incorrectly according 
to a spatially non-local segmentation model 

[Sect. 3.2, 1 st paragraph, especially lines 11-15. Note that the 
correctly and incorrectly classified pixels determine "positive" 
feedback map ( s ) and "negative" feedback map(s) (in the sense as 
described in the application), respectively.] 

14. Regarding claim 8, and similarly claim 32, Bhat further teaches/suggests 

• wherein the spatially non-local segmentation model defines spatially 
non-local observation characteristics of pixels that should have been 
excluded from one of the labeled groups 
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[Sect. 3.2, 1 st paragraph, especially lines 11-15. Note that 
classifying into a labeled group such as "people" inherently excludes 
pixels that do not belong to this group] 



15. Regarding claim 9, and similarly claim 33, Javidi further teaches/suggests 



• wherein a pixel -level historical model is updated at pixels identified 
as being incorrectly segmented 
[Col. 6, lines 47-55] 



*** 



1 6. Claims 1 0, 1 1 , 34 and 35 are rejected under 35 U.S.C. 1 02(b) as being 
unpatentable over Grimson et al. (Using Adaptive Tracking to Classify and Monitor 
Activities in a Site," Proceedings of IEEE Conference on Computer Vision and Pattern 
Recognition, 23-25 June 1998, pp. 22-29]), Bhat et al. ("Motion Detection and 
Segmentation Using Image Mosaics," IEEE International Conference on Multimedia and 
Expo, Vol. 3, 30 July -2 Aug. 2000, pp. 1577-1580) and Javidi (US 5,699,449) as 
applied to claims 6-9 and 30-33, and further in view of Goldenthal et al. (US 5,625,749). 



1 7. Regarding claim 1 0, and similarly claim 34, the combined invention of Grimson, 
Bhat and Javidi discloses all limitations of its parent, claim 9. 

The combined invention of Grimson, Bhat and Javidi does not expressly disclose 

• maintaining a per-pixel inclusion error model of pixel observations 
associated with occurrences of incorrect segmentation labeling 
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However, Goldenthal teaches/suggests maintaining an error model for each data point 
(a phone, or phoneme, in this case). [Col. 7, lines 33-37.] 

The combined invention of Grimson, Bhat and Javidi is combinable with Goldenthal 
because they have aspects that are from the same field of endeavor of 
segmentation/classification. 

At the time of the invention, it would have been obvious to one of ordinary skill in the art 
to modify the combined invention of Grimson, Bhat and Javidi with the teachings of 
Goldenthal by maintaining an error model at each pixel. The motivation would have 
been to assist in the classification process, as Goldenthal indicates in Col. 7, 
lines 41-44. 

Therefore, it would have been obvious to combine Goldenthal with Grimson, Bhat and 
Javidi to obtain the invention of claim 10. 

1 8. Regarding claim 1 1 , and similarly claim 35, Grimson discloses 

• the per-pixel inclusion error model corresponding to individual pixels 
include respective mixtures of Gaussian distributions 
[P. 23, left column, lines 3-8] 



*** 
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19. Claims 12 and 36 are rejected under 35 U.S.C. 102(b) as being unpatentable 
over Grimson et al. (Using Adaptive Tracking to Classify and Monitor Activities in a 
Site," Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 
23-25 June 1998, pp. 22-29]), Bhat et al. ("Motion Detection and Segmentation Using 
Image Mosaics," IEEE International Conference on Multimedia and Expo, Vol. 3, 30 July 
-2 Aug. 2000, pp. 1577-1580), Javidi (US 5,699,449) and Goldenthal et al. (US 
5,625,749) as applied to claims 10, 11, 34 and 35 above, and further in view of Tresp et 
al. (US 6,272,480). 



20. Regarding claim 12, and similarly claim 36, the combined invention of Grimson, 
Bhat, Javidi and Goldenthal discloses all limitations of its parent, claim 11 . 

The combined invention of Grimson, Bhat, Javidi and Goldenthal does not expressly 
disclose 

• wherein updating a pixel -level historical model comprises merging a per- 
pixel historical model and a per-pixel inclusion error model 

However, Tresp teaches/suggests merging a model (a neural network modeling a 
dynamic system in this case) with a corresponding error model. [See Fig. 3; Col. 3, 
lines 45-53.] 

The combined invention of Grimson, Bhat, Javidi and Goldenthal is combinable with 
Tresp because they have aspects that are from the same field of endeavor of modeling. 
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At the time of the invention, it would have been obvious to one of ordinary skill in the art 
to modify the combined invention of Grimson, Bhat, Javidi and Goldenthal with the 
teachings of Tresp by merging a model with its corresponding error model. The 
motivation would have been to because in doing so only few measured values of the 
influencing variables (e.g., selected number of video frames) are needed for the 
construction of the model as Tresp indicated in Col. 1, line 45 - Col. 2, line 9. 



Therefore, it would have been obvious to combine Tresp with Grimson, Bhat, Javidi and 
Goldenthal to obtain the invention of claim 12. 

*** 

21. Claims 15, 22 and 23 are rejected under 35 U.S.C. 102(b) as being unpatentable 
over Grimson et al. (Using Adaptive Tracking to Classify and Monitor Activities in a 
Site," Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 
23-25 June 1998, pp. 22-29]), and Bhat et al. ("Motion Detection and Segmentation 
Using Image Mosaics," IEEE International Conference on Multimedia and Expo, Vol. 3, 
30 July -2 Aug. 2000, pp. 1577-1580) as applied to claims 1-5, 13-14,24-29, 37-38 
and 45, and further in view of Gordon et al. (US 6,661,918). 

22. Regarding claim 15, the combined invention of Grimson and Bhat discloses all 
limitations of its parent, claim 14. 



Application/Control Number: 10/006,687 



Art Unit: 2625 



Page 12 



The combined invention of Grimson and Bhat does not expressly disclose 

• wherein one or more of the feedback maps are generated based at least in 
part upon depth information or stereo disparity information, or both 

However, Gordon teaches/suggests using range information for segmentation an image 
into foreground (e.g., a face) and background. [See Fig. 2, refs. 22, 26.] 

The combined invention of Grimson and Bhat is combinable with Gordon because they 
have aspects that are from the same field of endeavor of segmentation/classification. 

At the time of the invention, it would have been obvious to one of ordinary skill in the art 
to modify the combined invention of Grimson and Bhat with the teachings of Gordon by 
using range information for segmentation. The motivation would have been because 
range-based data is largely independent of color image data and hence not adversely 
affected by the limitations associated with color-based segmentation, as indicated by 
Gordon in Col. 3, lines 19-23. 

Therefore, it would have been obvious to combine Gordon with Grimson and Bhat to 
obtain the invention of claim 1 5. 

23. Regarding claims 22 and 23, Gordon further discloses 

• (claim 22) wherein one or more pixel-level historical models incorporate 
per pixel depth information or stereo disparity information, or both 
[Fig. 2, refs. 22, 26] 
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• (claim 23) wherein pixels are segmented based at least in part upon 
depth information or stereo disparity information, or both 

[Fig. 2, refs. 20-26; Col. 4, lines 12-64] 

*** 

24. Claims 16-21 and 39-44 are rejected under 35 U.S.C. 102(b) as being 
unpatentable over Grimson et al. (Using Adaptive Tracking to Classify and Monitor 
Activities in a Site," Proceedings of IEEE Conference on Computer Vision and Pattern 
Recognition, 23-25 June 1998, pp. 22-29]), and Bhat et al. ("Motion Detection and 
Segmentation Using Image Mosaics," IEEE International Conference on Multimedia and 
Expo, Vol. 3, 30 July -2 Aug. 2000, pp. 1577-1580) as applied to claims 1-5, 13-14, 24= 
29, 37-38 and 45, and further in view of Owechko et al. (US 6,801 ,662). 

25. Regarding claims 16-17, and similarly claims 39-40, the combined invention of 
Grimson and Bhat discloses all limitations of its parent, claim 1 . 

The combined invention of Grimson and Bhat does not expressly disclose 

• (claim 16) generating one or more confidence maps associating pixels 
with respective measures of segmentation accuracy 

• (claim 17) merging multiple confidence maps to produce a merged 
confidence map 



However, Owechko teaches/suggests generating confidence maps with respective to 
different classifiers and fusing the confidence values together. [Fig. 1 ; Col. 4, lines 49- 
58.] 
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The combined invention of Grimson and Bhat is combinable with Owechko because 
they have aspects that are from the same field of endeavor of 
segmentation/classification. 

At the time of the invention, it would have been obvious to one of ordinary skill in the art 
to modify the combined invention of Grimson and Bhat with the teachings of Owechko 
by generating confidence maps with respective to different classifiers and fusing the 
confidence values together to make a final decision. The motivation would have been 
because in this way "votes" from different classifiers are combined in an optimal manner 
to result in a high accuracy in the final classification results, as indicated in Col. 4, lines 
59-62 of Owechko. 

Therefore, it would have been obvious to combine Owechko with Grimson and Bhat to 
obtain the invention of claims 16 and 17. 

26. Regarding claims 18-19 and 41-42, they are rejected because it is well known in 
the art to use real-valued confidence values (e.g., the outputs of the Hausdorf template 
matching in Fig. 1, ref. 155 of Owechko) and to combine various confidence values by 
addition (for its simplicity and ease of implementation), as well as to apply threshold to 
obtain the final results. 



27. 



Regarding claims 20 and 43, Owechko further discloses 



Application/Control Number: 10/006,687 Page 15 

Art Unit: 2625 

• wherein each of the confidence maps is generated based at least in part 
upon one or more of an image region analysis, a frame-wide image 
statistics analysis, or an analysis of the object or event content of 
the video frames 

[Fig. 1, refs. 105, 110, 120, 130, 140] 

28. Regarding claims 21 and 44, Grimson further discloses 

• wherein a pixel -level historical model includes a mixture of Gaussian 
distributions of pixel observations 

[P. 23, left column, lines 3-8] 



Conclusions and Contact Information 



29. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure: 

• Darrell et al. (US 6,188,777) - Uses range and other information for 
foreground/background segmentation 

• Wilcox et aL (US 6,072,542) - Uses Hidden Markov Model for video 
segmentation 

• Brumitt (US 6,658,136) - Models background using (mean, standard deviation) 

• Lipton et al. (US 6,625,310) - Builds per-pixel background models 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Yubin Hung whose telephone number is (703) 305- 
1896. The examiner can normally be reached on 7:30 - 4:00. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh Mehta can be reached on (703) 308-5246. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Yubin Hung 
Patent Examiner 
November 3, 2004 

BHAVESH M. MEHTA 
SUPERVISORY RATFNT EXAMINER 
TECHNOLOGY CENTER 2600 




